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1 Introduction 

The Department of State (DOS) Data Center Consolidation (DCC) program is an on-going 
enterprise-level program managed by the Bureau of Information Resource Management (IRM). 
This program was established in 2002 to support the Department's global mission of providing 
"anytime/anywhere" access to information systems via consolidated infrastructure. In the 
interest of ongoing spending reduction efforts across the Federal government, this program has 
served to eliminate duplication of data center services and infrastructure while controlling or 
reducing energy and real estate costs without compromise to the goal of a highly sustainable IT 
infrastructure service. 

The specific program goals have been, and continue to be, improved data center and server 
availability, scalability, security, sustainability, and cost containment. The program focuses on 
consolidating servers, data centers, and data center services. The overall scope of the effort is 
both Continental United States (CONUS) and Outside the Continental U.S. (OCONUS). These 
represent very different views of the world and are managed by two separate but integrated IT 
Resource Management initiatives within DOS: Global Information Technology Modernization 
Program (GITM) and Enterprise Server Operations Center (ESOC). To control duplication of 
effort, these two groups work closely and share workload in a way that supports centers of 
excellence or specialties within each group. GITM is the enterprise-wide lifecycle replacement 
program for the Department IT infrastructure addressing both CONUS and OCONUS sites. One 
aspect of the GITM program is the virtualization of the multiple IT infrastructure servers at 
OCONUS posts onto a local virtual infrastructure. The Department's Data Center Consolidation 
Program ESOC has focused on consolidating the enterprise-level CONUS data centers, seeing 
these as having the most immediate need thereby yielding the most substantive, immediate 
benefit. Therefore, throughout this paper we will address mainly the more recent CONUS or 
ESOC effort. Where elements of the GITM program contribute to the overall Department 
consolidation, they will be addressed and highlighted. 

1.1 Domestic Data Center, Server, Service Consolidation 

To achieve data center consolidation goals, the Department will consolidate CONUS servers into 
four data centers by the end of FY2015. Production servers will be located in two Tier III ESOCs 
- ESOC East and ESOC West - geographically separated and on separate power grids. Two other 
data centers in the Washington DC area will be repurposed: one for development systems and 
the other for a small number of specialized systems. 

Server and service consolidation is occurring as an integral part of merging data centers. The 
Department is simultaneously pursuing cost avoidance and reduction through server 
virtualization, adoption of more efficient server hardware platforms, and standardization and 
automation of the processes used to support servers. 

1.2 Overseas Network, Server, and Service Consolidation 

In addition to the ESOC DCC program, which focused on consolidation of our CONUS enterprise 
data centers, the Department has completed a multi-year effort by the GITM program to 
standardized infrastructure servers and desktops at DOS OCONUS locations (e.g. embassies and 
consulates). This effort followed on the heels of a DOS program to improve and consolidate 
management of the worldwide intranet. 

This GITM effort has now moved on to consolidating servers at foreign posts onto a virtual 
infrastructure. Multiple stand-alone will be replaced by a virtual infrastructure that will 
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consolidate servers onto an integrated platform. This effort is leveraging the server 
virtualization expertise gained through the ESOC DCC server consolidation effort. 

1.3 DOS IT Strategy, Cloud Computing, and Consolidation 

The DOS IT strategic goals are designed to support the dual Department objectives of 
Worldwide Diplomacy and American Citizen Services. As the various dimensions of cloud 
computing take shape over the next few years, the overall cloud model will function as both a 
driver and a catalyst to further consolidation. The Department's IT Goals specifically serve both 
to direct and to energize the domestic and overseas efforts. 

• Goal 1, Digital Diplomacy - applying modern tools of social networking and value-added 
information resources to the challenges of diplomacy and development. Digital Diplomacy 
will enable enhanced collaboration and information sharing among our internal and external 
stakeholders. 

• Goal 2, Cloud Computing - providing global access to all IT systems and information via a 
robust, worldwide, web-based infrastructure. State's cloud will include the Foreign Affairs 
Network, an evolving and growing platform capable of supporting all U.S. agencies operating 
overseas under Chief of Mission authority. 

• Goal 3, IT Leadership - ensuring effective governance of all IT resources, domestic and 
overseas, and focusing on accountability to our customers for excellence in performance 
and service delivery. 

As noted in Goal 2, the Department is incorporating the concepts of cloud computing into its 
support systems. Current work is focused primarily on Infrastructure as a Service (laaS), but the 
Department has begun the planning required to improve and enhance efforts in the areas of 
Platform as a Service (PaaS) and Software as a Service (SaaS). 

1.4 FDCCI Data Verification, Data Limitations 

The baseline asset inventory is based on a survey of data center managers, initially conducted in 
2007 and validated in 2010. As each data center moves from Stage 3: Migration Planning and 
into Stage 4: Migration Execution, the DCC program management office begins physical 
inventories on a monthly basis. This inventory primarily serves as a program management 
metric to ensure that we are on track for transitioning servers from the data center in time to 
support the decommissioning of the room, but it also serves as a very accurate verification of 
the inventory. 

The data in the inventory was generally gathered from the primary source. Where data was not 
available, estimates and averages were used. Estimates were used for the size of the server 
closets and number of servers at overseas sites. DOS also used estimates for real estate cost per 
square foot: as most of our data centers are in government-owned facilities in urban areas, we 
based the cost on the average rental rate for Class A office space in that location. 



2 Department Goals for Data Center Consolidation 

Many of the enterprise applications and network services for supporting the Worldwide DOS 
objectives are enabled by more than 3,500 servers residing in 12 data centers in the United 
States. Continuing its current efforts over the next four years, the ESOC DCC program plans to 
consolidate, optimize, and repurpose or decommission more than 70% of the data centers 
currently in operation to achieve the goals described in the table below. 
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Overseas the servers are housed at most embassies and consulates in small server rooms or 
closets. The current efforts for that community will be to virtualize the key infrastructure 
servers required at post. That effort will take a minimum of four years which is the average 
lifecycle to refresh all posts worldwide. Over time, the Department will be assessing the benefit 
of overseas regionalization. Regionalization, a program to establish regional services that 
reduces the Department's overseas server footprint and improves information security, will 
parallel the ESOC DCC effort in form and function. At the present time, however, bandwidth 
costs and the connectivity quality/reliability are pervasive roadblocks to full scale worldwide 
regionalization and/or domestic centralization. 

The benefits of both of these consolidation efforts, CONUS and OCONUS, are highlighted below 
in Table 1 DCC Program Goals. Our response to the FDCCI Strategic Objective to Reduce Cost is 
addressed in the Department's DCC Program Goal to Improve cost containment. Additional 
costs saving goals are addressed in Appendix B in the Planned Program Cost savings table. 
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Table 1 DCC Program Goals 



Federal DCCI 

Strategic 

Objective 


Department DCC 
Program Goal 


Department DCC Program 
Focus Areas and Objectives 


Quantitative Targets 


Qualitative Benefits 


Reduce Cost 


1. Improve cost 
containment 


• Dora Center, Server and Services 
Consolidation 

• Reduce/Optimize energy use 

• Reduce operational costs per server 

• Limit long-term capital investments 

• Process Automation 


• Number of Department domestic 
data centers reduced to four 

• All data centers operating at a Power 
Usage Effectiveness (PUE) range of 
1.3-1.6 

• Staffing remains relatively constant 
at the 2012 level through 2014 while 
consolidation efforts continue 

• Imnlpmpnt resource accounting 
system by 2014 


• Demonstrating fiscal 
responsibility and cost 
efficiency 

• Greater funding focus on 
Department's mission 
activities of Diplomacy 
and Citizen Services 


Reduce 

Environmental 
Impact 


2. Improve 

environmental 
sustainability 
of operations 


• Dora Center and Server Consolidation 

• Reduce power consumption per unit of 
processing capacity 

• Optimize cooling, power distribution, 
monitoring 

• Achieve Data Center-related goals set forth 
in Agency Sustainability Plan, Goal 9: 
Electronic Stewardship and Data Centers. 


• 70% of servers hosted on virtual 
infrastructure by 2015 

• ESOC West accredited at the 
Leadership in Energy and 
Environmental Design (LEED) silver 
level 

• 25% of posts critical server 
infrastructures virtualized - Full 
deployment in 2015 


• Reduced power 
consumption and green 
house gas emissions 

• Reduced cost 

• Reduced power 
consumption in all 
OCONUS DOS posts 
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Federal DCCI 

Strategic 

Objective 


Department DCC 
Program Goal 


Department DCC Program 
Focus Areas and Objectives 


Quantitative Targets 


Qualitative Benefits 


Improve 
Efficiency & 
Service Levels 


3. Improve 
operational 
efficiency 


• Data Center Services Consolidation 

• Implement Information Technology 
Infrastructure Library (ITIL)-based 
processes to support breadth of ESOC 
operations 

• Implement an Enterprise Service 
Management tools suite to support and 
measure ESOC service processes 


• All systems monitored for availability 
through common system 

• 98% of server outages escalated to 
system owner within 10 minutes 

• 70% of systems using common 
backup solution 

• System configuration fully managed 
under ITIL-based processes 

• Consolidated team supporting both 
CONUS and OCONUS virtual 
infrastructure 

• Reduced hardware inventory 
overseas resulting in lower shipping 
costs 


• Higher and consistent 
level of service provided 
to customers 

• Reduced cost for higher 
level of service 

• Improved security 
through fewer, more 
standard platforms 

• Centralized team provides 
center of excellence for 
support 


Enhance 
Business 
Agility 


4. Improve IT 
infrastructure 
services 


• Data Center, Server and Services 
Consolidation 

• Expand ITIL management practices to 
measure customer satisfaction 

• Grow cloud computing across the three 
service areas: laaS, PaaS, and SaaS. 


• Virtual Systems provisioned on 
demand within 24 hours 

• Support for private, partner, and 
public cloud services as required 

• Implement continuous feedback loop 
with customers that reports 
customer satisfaction to State 
governance authority 

• Use of virtual infrastructure overseas 
will be expanded in time to support 
limited local deployment of virtual 
infrastructure. This will bring laaS 
directly to remote locations 


• More responsive service 
provided to customers 

• Better insight into service 
provided to customers 
and achievement of 
service level agreements 

• Faster and better service 
to American citizens 

• Sharing of processes and 
information across 
Government Agencies 
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3 Implementing Shared Services/Multi -tenancy 

The DOS has numerous initiatives focused on providing shared services/multi-tenancy. The Department has a long history of providing services 
to separately funded organizations as well as other government agencies. On the domestic side, the Department has numerous key initiatives in 
play designed to consolidate service and support. These initiatives range in cost, duration, and state of completion. The purpose of this section 
is to broadly introduce the existing initiatives and then turn our attention to the primary purpose of the paper: data center, server, and services 
consolidation. Table 2 provides an overview of each of the programs, the goals of the program, stakeholders, and the general quantitative and 
qualitative delivery expectations as currently established. 



Table 2 Department Shared Services Programs 



Program 


Goals 


Stakeholders 


Quantitative Benefits 


Qualitative Benefits 


IT Desktop 
PorKnliHatinn 

• W 1 1 JUI IUu LIUI 1 


Centralize domestic desktop 
rn^nappmpnt inrlnHinp file 
print, e-mail, and desktop 
operating system 


Domestic DOS users 


Ultimately, reduced cost per desktop 


Centralized management 
Standard desktop managed 
environment - more secure 


FAN 


Foreign Affairs Network (FAN) 
supporting multiple agencies 
on Department OpenNet 


All agencies that 
interact with DOS at 
post 


Reduced network costs for other 
agencies operating overseas 


Improved worldwide network service 


Regionalization 


Establish overseas regional 
services 


Qualified Overseas 
Posts 


Reduces server footprint and power 
consumption 


Improved availability and security, 
better response, and overall reduced 
bandwidth cost 


GITM 

Virtualization 
Program 


Global Modernization of 
Overseas Infrastructure 


Overseas Posts 


Reduced server footprint and power 
consumption 

Reduced desktop footprint as desktop 
virtualization program begins 
Improved sustainability 


Centralized management of overseas 
server infrastructure as a service 
Improved patch and security 
management at high risk posts 


Security Services 


Establish centralized network 
perimeter and enclave security 


Worldwide OpenNet 
community 


Reduce number of individual Local 
Area Networks (LANS) that require 
physically separate networks 


Improved access across agency 
through shared infrastructure 
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Program 


Goals 


Stakeholders 


Quantitative Benefits 


Qualitative Benefits 


SMART 


Centralized messaging 


Worldwide Users of 
DOS Cable system 


Reduced post hardware and software 


Improved availability 
More feature rich system 
Central management 


Data Center 
Consolidation 


Consolidate domestic data 
centers, servers, and 
enterprise data center services 


Domestic 
Infrastructure and 
server owners 


Reduced footprint, particularly in high 
rent district of Washington, DC 
Managed Power and improved power 
utilization 


Centralized server support for laaS 
More scalable and secure 


Application Level 
Shared Services 


Provided support for 
Enterprise Microsoft Office 
Sharepoint Server (MOSS) 
implementation as well as 
local post shared 
administrative tools 


Worldwide DOS Users 


Central support for key department 
operational data 


Improved services for data 
management 


Security 
Management 


Centralize management and 
reporting of worldwide 
windows network 


Worldwide DOS Users 


Reduced cost through centralized 
patch distribution, intrusion detection 
system (IDS), and antivirus 
Reduced vulnerability at the desktop 
level demonstrated by reduced 
network intrusions 


Improved patch and security 
management worldwide 
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For purposes of the FDCCI, the primary focus of this planning document is the CONUS Data 
Center Consolidation. The Data Center Consolidation effort works in concert with each of these 
programs to ensure that Data Center, Server, and Services consolidation potential is realized in 
each program. The following chart outlines the shared program services that the Department 
DCC has with each of these separate programs. 



Table 3 Shared Services Alignment to DCC Program 



Program 


ESOC DCC Alignment 


IT Desktop 
Consolidation 


Data Center Consolidation provides centralized housing for servers such as file, 
print, e-mail, Dynamic Host Configuration Protocol (DHCP), messaging, and active 
directory 


FAN 


As the FAN expands to include more agencies, the possibility of hosting partner 
systems either domestically or overseas will be assessed. Currently this initiative is 
predominantly focused on network consolidation 


Regionalization 


As regional centers are established, the ESOC DCC program will provide support for 
virtualization 


GITM 

Virtualization 
Program 


Global Modernization of Overseas Infrastructure, Server Virtualization 


Security Services 


Establish centralized network perimeter and enclave security 


SMART 


This government wide messaging system is housed within the DCC ESOC and makes 
use of the virtual infrastructure provided by the DCC 


Application Level 
Shared Services 


Domestic servers that support these shared application systems are housed in the 
DCC ESOC. Overseas systems will be migrated to the GITM Virtual Infrastructure as 
it rolls out over the next four years 


Security 
Management 


All systems associated with this effort are housed within the DCC ESOC data centers 
and make use of the virtual infrastructure 



Within this context, data center, server, and enterprise data center services needed to support 
these goals are provided by the ESOC DCC in support of the programs noted above. For 
instance, currently the Department houses systems from multiple agencies. In addition, the 
Department DCC program provides support for access to community networks. Through these 
shared government networks, the Department is able to share information amongst the larger 
U.S. federal community. This arrangement as well as other shared services initiatives will be 
enhanced through the combined FAN and DCC programs. The direction to expand and grow 
both network and data center services to the broader federal community as required by the 
FDCCI is in concert with the Department's 2010 IT Strategic Plan and is actively being 
implemented by the Department's programs noted in this section. 

Each of these shared services programs is a large and complex set of projects all moving forward 
in their own tracks but governed by the strategic plans noted above. The purpose of this paper 
has been to address in detail only the Data Center Consolidation portion. To that end, further 
elaboration on the other programs will be limited to those areas where there is a direct bearing 
on the ESOC DCC plan. 
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4 Department Approach, Rationale and Timeline 

The Department's CONUS data center consolidation strategy is to consolidate all domestic data 
centers into four enterprise data centers. Figure 1 presents the current Master Plan Schedule 
for the program that is used to manage the DCC program. The first four rows are the "target" 
data centers. The fifth row contains the "source" data centers that will be consolidated and 
decommissioned. For completeness, rows six and seven add the Department's plans for 
OCONUS and other smaller CONUS sites. 

The Department's domestic strategy - two geographically dispersed production data centers 
plus two additional repurposed data centers for development networks and specialized systems 
- will position the Department for the next 10 years, at a minimum. Consolidation of services, 
including virtualization at OCONUS sites, will allow the Department to mature service delivery 
and support processes and monitor and manage the true costs of data center IT services. Once 
established, the production data centers will provide the highly available, highly accessible data 
centers required to host cloud computing services for the Department and for other Foreign 
Affairs agencies. 

Figure 1 Master Plan Schedule 
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The Master Plan Schedule is designed to highlight four elements of the data center 
consolidation effort across the program time line: capital equipment purchases (500 systems, 40 
Gb network upgrade, etc.), service offerings (DMZ support, Development LAN), 
decommissioning strategy (HST 5440 in FY13), and end state capacity/service orientation 
(Intranet, Development LAN, etc). These are planning guidelines used for resource and budget 
planning. These four elements and the timing are assessed annually and adjustments will be 
made. This allows the program to proceed in a progressively elaborated fashion, allowing mid- 
program adjustments as unforeseen challenges are encountered. Within this plan and not 
displayed but noted previously, Data Center Consolidation is simultaneously pursuing two 
supporting strategies: server consolidation and data center services consolidation. 

The server consolidation component seeks to reduce physical servers by merging like servers 
and consolidating these and all other systems into an enterprise virtual infrastructure. This 
strategy reduces unused capacity and total cost of ownership by lowering power costs, server 
hardware costs, acquisition costs and the data center footprint along with their related 
environmental impacts. The Department plans to virtualize 50% of servers over the next three 
years, with a longer-term goal of 70% server virtualization. Services consolidation is designed to 
address the need for common services, such as backup, once servers are housed in centralized 
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facilities. The IT Services Consolidation effort is an ongoing part of the overall data center 
consolidation program. 

As the program began, two primary alternative consolidation cost benefit analyses were 
conducted by the DCC Program Management Office (PMO): the first to determine the overall 
DCC requirements and strategy; the second to develop an acquisition strategy for ESOC West. 

The first study was the U.S. Department of State Business Case Analysis, Enterprise Data Center 
Strategy Study, 20 March 2008. The purpose of the study was to conduct a business case 
analysis to support the development of a data center consolidation strategy. The 
recommendation of the study was that the Department's servers be transitioned into two 
geographically dispersed Tier 3 commercially owned facilities. The intent was to improve overall 
service and reduce delivery time by obtaining services from an established vendor. This is the 
strategy that the Department embarked on in late 2008. The goal was to deliver the first 
commercially supplied service in 2009 and the second in 2012. DOS also chose to retain a third 
existing facility in the Washington, DC area to function as an alternative site. The first site, ESOC 
East went production in 2009. 

After the first site was installed, the Department was challenged with the issue of preparing for 
the second site in 2012. At that time, the American Recovery and Reinvestment Act (ARRA) 
became available as a possible option to build a federally owned data center in the necessary 
time frame. This led to a second smaller study which examined options for the acquisition of an 
ESOC West Data Center. A number of strategies were explored, including the use of military 
properties excessed via the Base Realignment and Closure (BRAC) process, obtaining a 
commercial service again like ESOC East, purchasing land, or the Department leasing land. After 
an analysis, the final strategy of leasing land on a GSA-owned campus was selected. The 
strategy best achieved the overall ARRA objectives, including near-term job creation, and 
supported the Green Data Centers initiatives as articulated in the Agency Sustainability Plan. A 
location was selected that offered geographic and power-grid diversity, communications 
infrastructure, ideal climate conditions, minimum concern for natural disasters, and cost 
effective power. As the ARRA program came to fruition, this provided needed funding for the 
Department to set the plans in place to pursue the two additional Washington, DC based data 
centers that would house development networks and specialized systems, thereby completing 
the overall high level data center strategy. 

The primary OCONUS consolidation activity is the consolation of physical servers at post onto a 
virtual server infrastructure. Each OCONUS site will be served by a local, integrated platform 
located within the site's small server room or closet. The virtual server platform will be 
implemented at all OCONUS sites. 

This effort is being implemented by GITM program, which is the Department's enterprise-wide 
lifecycle replacement program for IT infrastructure. The virtualization program began in FY11 
and will take four years to implement at all OCONUS sites. 

The goal of the GITM OCONUS virtualization project is reduce server footprint and power 
consumption, improve sustainability, and streamline server support processes. 

5 Department Governance Framework for Data Center 
Consolidation 

The DCC effort is managed by the DCC PMO in the System Integration Office (SIO) of the IRM 
Bureau. The DCC PMO uses Project Management Institute (PMI)-based project management 
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methods for initiating, planning, executing, controlling and monitoring, and closing projects. 
Under the umbrella of the DCC program, a number of concurrent projects are being executed at 
any given point in time. 

The DCC PMO uses three layers of transition planning, and out of these processes, consensus is 
reached regarding the disposition of each system and each data center. 



Transition Plan 


Description 




Data Center 
Transition Plan (DTP) 


Overarching projects to migrate all systems out of 
a data center and either decommission or 
repurpose that data center. Vertical planning to 
address the consolidation of all components 
within a data center. 


Customer Transition 
Plan (CTP) 


Projects to analyze, plan, and oversee the 
execution of migrations of the full range of 
systems for a particular customer. Horizontal 
planning to ensure customer needs / service 
levels are addressed. 


System Transition 
Plan (STP) 


Project to plan, execute, and monitor a specific 
system's transition to a new data center. Tactical 
process to ensure each transition is well managed 
and efficient. 



Figure 2 Transition Planning Processes 




The DCC program maintains metrics at multiple levels: system and process performance. 
Process performance level metrics are gathered from across the program, measuring 
achievement towards strategic and tactical goals and ongoing operational efficiencies. System 
performance metrics include the Response Times of critical applications (such as Microsoft 
Office SharePoint Server) and the CPU and Storage utilization of the Virtual Infrastructure. 
Process performance metrics maintained include the time-to-process virtual server requests and 
the accuracy of Virtual Machine Request Forms. 

An Information Technology Infrastructure Library (ITIL)-based Enterprise Service Management 
tool suite is being implemented to support ESOC operations. This suite will streamline 
configuration and change management and will enable the ESOC to more accurately allocate 
costs and report service levels provided to ESOC customers. The tool suite will also enable ESOC 
managers to identify areas for improvement to achieve greater operational efficiencies. 

System level statistics are also addressed, including energy consumption and power usage 
effectiveness (PUE). These statistics are used to estimate environmental impacts such as 
reduced greenhouse gas emissions. The metrics are gathered weekly for internal program 
consumption and published monthly to a wider audience that includes senior management. 

Department-level IT governance is provided by the E-Government Program Board (E-Gov PB). 
This board provides senior-level guidance to the Under Secretary for Management that 
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addresses the full range of Department E-Gov and IT investment portfolio and project 
management activities. The E-Gov PB has three primary purposes: 

1. Prioritize and resolve significant policy, strategic and resource issues concerning the 
Department's investments in E-Gov and IT initiatives. 

2. Drive innovation in the use of technology while effectively managing E-Gov/IT capital 
decisions; and 

3. Ensure systematic selection, control, and evaluation of all the Department's E-Gov/IT 
programs and investments, as required by law and OMB mandates. 

The E-Gov PB provides an enterprise wide investment review, decision making, and governance 
process, which maximizes the value and mitigates the risk associated with complex, central, de- 
central, and concurrent information technology initiatives and acquisitions. 

5 . 1 Cost-benefit Analysis 

5.1.1 CB A App roach 

The DOS entered into the Data Center Consolidation plan with in anticipation of the fact that the 
consolidation would have significant front-loaded costs but when executed properly will provide 
superior service and cost avoidance or reduction over time. Given the dynamic nature of the 
task, the industry, and federal funding cycles, the budget and expected cost model must be 
approached with a view toward adjustments. To that end, the program has been designed to 
invest in capital improvements, including buildings, servers, and services that will allow growth 
through a scalable planning and implementation approach. 

The plan to address the variable nature of this program included a threefold approach to cost 
containment and possible reduction: 

1. Cost Avoidance - this was the first focus and materialized primarily through 
virtualization. 

2. Cost Control - Much of the cost of the data centers was "hidden" through various 
distributed organizations. The program has sought through centralization of costs and 
resource accounting to obtain an accurate picture of the true cost of the service. The 
DCC is preparing to implement a cost allocation system that will clearly allow the DOS to 
evaluate the service delivery on a cost basis. 

3. Cost reduction - Over time the ability to both control and possibly reduce costs will be 
achieved in the out years as operations assumes steady state and support activities can 
be streamlined. 

The budget for this architecture has been expressed in terms of the table below which presents 
the Operating Expenses (OPEX) and Capital Expenses (CAPEX) required to establish, refresh, 
operate, and maintain the Department's domestic enterprise data centers. The overseas 
component of consolidation is not covered in this section as that program has a separate cost 
structure in place and is funded accordingly. Improvements in that program are being 
considered separately. 

The domestic costs below are estimates used for analysis; with the exception of CY+1, these are 
not budget request numbers that have gone forward. 

5.1.2 CBA Budget 

As noted in the table below, the estimated data center OPEX costs increase over the coming 
years from $25M in FY10 to $32M in FY15. Currently, as utilization of the ESOCs expand and 
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higher service levels are achieved, the DCC estimates that the growth in total OPEX costs will 
hold to roughly the rate of inflation. The overall cost increase is a result of improved service 
levels in the new and more expensive Tier 3 data centers. This new level is required to achieve 
the SLAs required by DOS data center customers. In the original study, the need for improved 
and high availability data centers was identified as the ESOC Customers' number one priority. 
Specifically, the requirements identified in the original study were as follows. 

• Establish high availability data centers: systems transitioning from Tier 2 (or worse) 
data centers to enterprise-class Tier 3 data centers; 100% availability SLA for power and 
environmental; 24x7 ESOC support staff; high availability network design; geographically 
dispersed; high speed replication between sites 

• Maintain security: all servers will be hosted in secure facilities with 24x7 Diplomatic 
Security Uniformed Guards 

• Scalability: support future Department requirements, even if unforeseen, and do so in 
an acceptable time frame 

• Cost containment of a centralized data center budget; establish costs, develop a cost 
recovery system, control cost growth and reduce if possible 

The Budgeted Costs for End-State Target ESOCs includes the following CAPEX costs between 
now and FY15: 

• FY11: ESOC West, DevLan, and VI Buildout IT Infrastructure - $12M 

• FY12: ESOC West scale out and lifecycle maintenance - $8M 

• FY13: ESOC West 2nd Pod- $40M 

• FY14: ESOC East and West Network Core to 40 Gb - $10M 

• FY15: Virtual Infrastructure lifecycle refresh - $15M 

One of the critical elements of the program is the cost avoidance from the virtualization 
program. We estimate that the Department will be able to reallocate over $8M per year once 
the transitions are complete and steady state has been achieved at the ESOCs. The table below 
presents the cost avoidance through server virtualization. The source of our calculation was a 
"green calculator" that VMWare makes available to estimate costs savings associated with 
transitioning from physical to virtual servers. 

Table 4 DCC Cost Estimates show four general areas: end state cost estimate for the four 
domestic data centers, current OPEX for data centers to be decommissioned, total cost estimate 
(previous to added), and cost avoidance. The table includes ARRA funding as a CAPEX expense 
in FY09 and FY10. These costs are aggregate estimates of the full DCC costs, which span several 
organizations and funding sources. The costs do not directly reflect the Department's budget 
requests. 
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Table 4 DCC Cost Estimates 





Budgeted Costs for End-State Target ESOCs 


OPEX 


$ 


12,295 


$ 


22,789 


$ 


20,436 


$ 


28,367 


$ 


30,008 


$ 


30,908 


$ 


31,836 


CAP EX 


$ 


31,760 


$ 


102,712 


$ 


12,214 


$ 


7,900 


$ 


40,000 


$ 


10,000 


$ 


15,000 


Total 


$ 


44,055 


$ 


125,501 


$ 


32,650 


$ 


36,267 


$ 


70,008 


$ 


40,908 


$ 


46,836 


Estimated Costs for Data Centers to be Decommisionned 


OPEX (including Labor) 


$ 


6,531 


$ 


6,531 


$ 


6,375 


$ 


4,977 


$ 


3,892 


$ 


3,392 


$ 


779 


CAPEX (server lifecycle replacement) 


$ 


1,159 


$ 


1,159 


$ 


1,159 


$ 


1,101 


$ 


427 


$ 


257 


$ 


93 


Total 


$ 


7,690 


$ 


7,690 


$ 


7,534 


$ 


6,078 


$ 


4,319 


$ 


3,649 


$ 


872 


Total Data Center Costs 


OPEX 


$ 


18,827 


$ 


29,320 


$ 


26,812 


$ 


33,344 


$ 


33,900 


$ 


34,301 


$ 


32,614 


CAPEX 


$ 


31,760 


$ 


102,712 


$ 


12,214 


$ 


7,900 


$ 


40,000 


$ 


10,000 


$ 


15,000 


Total 


$ 


50,587 


$ 


132,032 


$ 


39,026 


$ 


41,244 


$ 


73,900 


$ 


44,301 


$ 


47,614 


Costs Avoided 


OPEX Costs Avoided via Decommisioning DCs 


$ 


$ 


$ 


156 


$ 


1,554 


$ 


2,639 


$ 


3,139 


$ 


5,753 


CAPEX Costs Avoided through Virtualization 


$ 


$ 


782 


$ 


1,477 


$ 


2,065 


$ 


3,539 


$ 


4,986 


$ 


5,817 



5.1.3 Sources of Funds 

The amounts noted above represent estimates in an attempt to show the costs in the three 
broad categories: OPEX, CAPEX, and DCC. Currently, ESOC is including requests for the majority 
of this funding in their annual requests. The major exception is FY12, which is included in the A 
Bureau (Facilities) budget. We are planning to begin a program of chargeback, but must plan it 
and roll it in such a way that bureaus can appropriately budget. The current proposed rollout 
plan is shown in Table 5 below. 



Table 5 - Cost Recovery Implementation Timeline 





FY10 FY11 




FY12 FY13 


FY14 


FY15 


FY16 


Funding Model 


CIF Funded 


Full Cost Allocation 


Implementation Lifecycle 


Planning 


Trial 


Production 


Deliverables 


Remedy Implementation 


Cost Reporting 


Collect Funds 



5.2 Risk Management and Mitigation 

An integral part of the PMI-based project management methods used by the DCC PMO is 
identification, analysis, reporting, and tracking of project risks. The risk management processes 
used by the DCC PMO are described in detail in the DCC Program Management Plan. An 
overview of these processes is described below. 

Project level risks are initially identified during Project Planning activities by the DCC PMO and 
Matrix Managers and are tracked via a standard Risk Register on each project's website portal. 

Project managers and the Matrix Managers of project teams continue to identify risks through 
the life of the project via evaluation of milestone progress and status, specifically identifying 
risks that impact cost, schedule, or quality. 

The objective of the Analysis activity is to convert risk data into decision-making information. In 
addition to a Description of each risk identified, the following characteristics are also assessed 
during the risk analysis phase: Likelihood, Impact, Trigger, Management Strategy, and Mitigation 
Strategy. Each of these characteristics is reported in the Risk Register, monitored as the project 
progresses and reassessed each week as part of the Project Status Reporting. 

Any risks which are raised to "High" (project approaching trigger) or "Critical" (risk is realized or 
is certain to be realized) are also highlighted through Risk-type Project Taskers. These taskers 
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are presented on the project's website portal and reviewed at each status meeting. These 
taskers also appear in that project's Dashboard on the Program Status Report. 

The DCC Program-wide Risk Register is shown on the following pages. 
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Table 6 DCC Program Level Risk Register 





Description 


Likelihood 


Impact 


Trigger 


Management Strategy 


Mitigation Strategy 




Funding not secured 
to continue to scale 
out data center 
infrastructure to 
support additional 
customers 


Medium 


High 


New FY Budget 
passbacks do not 
include requested 
CAPEX funds. 


1 . Continue capacity planning and budget 
forecasting to ensure that full CAPEX 
requirements are determined to support 
projected customer base. 

2. Work with IRM Strategic Planning Office 
to ensure that DCC CAPEX budget 

IcqUllclllclllo die lllUIUUcU in cdCN yedl b 

budget request. 


1. Set expectations in 
customer outreach 
meetings that DCC budget 
may not support future 
requirements 

2. Work with IRM Strategic 
Planning Office to 
accelerate plans to 
implement charge back 
program 




Costs exceed budget 
estimates / funding 
not secured to operate 
and maintain ESOCs 


Low 


High 


New FY Budget 
passbacks do not 
include requested 
OPEX funds. 


1 . Continue capacity planning and budget 
forecasting to ensure that full OPEX 
requirements are determined to operate 
and maintain ESOCs. 

£L. vvOiK Wltn Inlvl oTraTGylC rlannlny UTTICG 

to ensure that DCC OPEX budget 
requirements are included in each year's 
budget request. 


1. Work with IRM Strategic 
Planning Office to 
accelerate plans to 
implement charge back 
program 

2. Scale back ESOC 
services 




- 


ESOC West not 
opened to support 
customers by FY1 2Q3 


Medium 


Medium 


Design / Build project 
schedule or Core 
Infrastructure project 
plan projects that 
services will not be 
available on time 


1 . Monitor Design / Build schedule on a 
weekly basis to ensure that timelines are 
being met and risks are be monitored. 

2. Initiate ESOC West Core Infrastructure 
project in FY11Q4, integrating lessons 
learned from ESOC East Core 
Infrastructure project. 


1. Set expectations in 
customer outreach 
meetings regarding 
revised ESOC West 
schedule 

2. Use ESOC East and 
BIMC capacities to bridge 
gap in schedule 


Customer 

requirements grow to 
exceed ESOC 
capacities 


Low 


Medium 


Planned System 
Transitions (as 
projected via Customer 
Transition Plans) 
exceed ESOC 
capacities 


1 . Continue Customer Transition Planning to 
validate and refine ESOC capacity plans. 

2. Continue to advocate use of virtualization 
for all appropriate systems. 


1 . Proceed without 
documentation, initiate a 
later project dedicated to 
completing documentation 

2. Slip facility acceptance 
and transition dates 
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ID 


Description 


Likelihood 


Impact 




Management Strategy 


Mitigation Strategy 




System owners do not 
move into ESOCs in 
accordance to the 
plan and schedule 
(including externally 
hosted applications) 


Medium 


High 


Planned/Budgeted 

resources 

for migration 


1 . Step up market awareness 


1 . Reassess plan regularly 

2. Make appropriate 
adjustments and publish 




System requiring re- 
engineering to 

fi mnt'mn in 1 — PAP 

function in cbUO 
environment (latency 
and through put 
performance) 


Low 


Low 


Monthly performance 
report increase in 
measured data 


1 . Establish standard measurement 
techniques 

2. Make appropriate software reengineering 
recommendations 


1 . Establish fund and 
technical sources to 
accommodate 




Potential out year cost 
increases to move 
from a consolidated 
/virtualized 
environment to a 
cloud environment 


High 


High 


Customer feedback 


1. Budget accordingly 

2. Continue to evaluate the cost benefit and 
adjust outflows accordingly 


1. Monitor budget 




ESOCs are not 
meeting customer 
SLAs 


Low 


Medium 


Reported service 
levels fail to meet more 
than 50% of customer 
SLAs two consecutive 
months 


1 . Measure and report service levels 
provided on a monthly basis. 

2. Perform Problem Management to mitigate 
low service levels as discovered. 


1 . Establish ESOC-wide 
Problem Management 
tiger team to perform in 
depth analysis of why 
service levels 
agreements are not being 
met 

2. Either revise processes, 
request additional 
resources to met service 
level agreements, or 
revise SLAs 
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5.3 Acquisition Management 

The DCC PMO acquisition plan includes three primary components. Two components are the 
separate efforts to develop and execute strategies to acquire data center facilities for ESOC East 
and ESOC West. The third component is the ongoing acquisition of IT hardware, software, and 
services for the DCC program. 

The high-level acquisition strategy for ESOC East was developed as part of an analysis performed 
on behalf of the Department in 2007. This analysis recommended that the Department lease 
colocation services from a commercial data center provider to establish the first of two Tier 3 
enterprise data centers. This strategy was successfully executed in 2008 with the establishment 
of a long term contract with a leading data center services provider. 

A number of acquisition strategies were explored for ESOC West, including the use of a BRAC 
site, leasing a commercial service, purchasing land, and leasing land. After much analysis, the 
final strategy of leasing land on GSA-owned campus was selected. The Design/Build phase of 
the establishment of ESOC West has followed GSA procedures. The GSA Contracting Officer 
(CO) is responsible for the execution of the Design/Build contract. DOS and GSA are actively 
involved in monitoring the project in cooperation with the agency of Construction Management 
Authority. 

The third acquisition component of the DCC has been the ongoing acquisition of IT hardware, 
software, and services. A plan was developed for purchase of all IT purchases, including services 
to begin consolidation in anticipation of the new data centers. That plan included the cash 
outlays over the life of the implementation period and outlined the services and 
hardware/software that would be purchased. The Department's Office of Acquisitions 
Management was responsible for execution of the plan in conformance with the Federal 
Acquisition Regulations (FAR) and all Departmental policies and procedures. A part of that plan 
included the establishment of contracts providing enterprise licensing where appropriate. A 
foundation of the enterprise license structure has been through the use of GSA Schedule costs. 
Further cost reductions were negotiated based upon anticipated purchase amounts over the life 
of the consolidation efforts. 

The acquisition authorities have worked to ensure that small businesses were adequately 
represented in DCC acquisitions. The Department's and GSA's Acquisitions Offices, Legal, and 
the Office of Small and Disadvantaged Business Utilization worked closely with the program and 
project managers to ensure all goods and services requirements are met in accordance with the 
FAR and all Departmental policies and procedures, including the security aspects as identified in 
12FAM210. In addition to the established Department and GSA rules and regulations, there are 
additional "Recipient Reporting" requirement as specified by OMB. The RFP for the Design / 
Build of ESOC West followed specific guidelines that required the selected vendors to 
implement a small business plan where 30% of the project went to small business. This was 
successfully implemented. 

5.4 Communications Strategy 

The DCC Program uses a multi-tiered communication strategy to keep all levels of decisions- 
makers informed regarding all aspects of the DCC Program, including the timeline for 
establishment of the new data centers, ESOC services being offered, cost allocations, and risks. 

At the highest level are the E-Gov PB (described in Section 5) and Advisory Group (comprised of 
Bureau Executive Directors) which are critical in supporting and facilitating the successful 
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execution of this initiative. This forum will be used to ensure there is Department-wide 
participation in the enterprise initiative and that the requisite policy guidance is in place. 

At the level is the DCC Program's outreach to Bureau Executive Directors. The DOS Chief 
Information Officer (CIO) has fully authorized the Data Center Consolidation Charter as 
expressed further in the programs official charter. In anticipation of the project, a round of 
briefings was held at the initiative of the DCC effort to inform Executive Directors of the 
programs goals and timelines and to obtain their support of the program. As ESOC West comes 
close to being brought online, a round of updated briefings will be presented to Bureau 
Executive Directors. Interim updates have also been presented as required. 

The DCC Program keeps IRM management, including the CIO and the Deputy ClOs, abreast of 
DCC activities via a Weekly Activities Report (WAR) that details all significant program activities. 
IRM management and Resource Management decision makers are all also regularly briefed 
concerning financial status of the DCC program and upcoming budget requests. 

The primary communications vehicles to ESOC customers are ongoing Customer Transition 
Planning projects. As described elsewhere in this plan, these projects analyze, plan, and oversee 
the execution of migrations of the full range of systems for a particular customer. As with other 
DCC projects, weekly status meetings are held to plan system transitions and to keep customers 
abreast of DCC activities. 

Lastly, the DCC PM presents DCC overview briefings on a semi-annual basis at the weekly IRM 
Network Management Meetings. These briefings are open to all Department staff. These 
briefings are also recorded and can be streamed to desktop computers via BNET, the 
Department's internally broadcast television network. 

6 Progress 

6.1 FDCCI Consolidation Progress 

The Department is on track to meet the 2011 amended targets. We decommissioned one data 
center and are on track to decommission three additional data centers by the end of the year. 
There is one data center that we will not be able to close as planned in 2011. Due to the 
inability of one key customer to transition a critical system, the closure of this data center has 
been moved to 2012. The Department informed OMB and GSA of this schedule change in July 
2012. 

The Department is on track to meet our 2012 targets. Most significant to the DCC effort is that 
we will close two data centers. 

We remain on schedule for completion of our DCC effort in 2015: 

• Our east coast data (ESOC East) has been operational since mid-2009: we have consolidated 
over 2,200 servers into it. 

• Over 50% of the servers at ESOC East are virtual machines on our enterprise virtual 
infrastructure, exceeding the 20% target that we set for ourselves in our Sustainability Plan. 

• Our western data center (ESOC West) is under construction and are on track for full 
production operations in mid-2012. It will be a LEED Gold certified facility. 

• Together, these two Tier III data centers will establish a highly-available, scalable, and 
sustainable environment that will support the Department for over 20 years. 
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Our greatest success as a program is that we have been able to support over 2,200 servers in a 
modern, reliable, secure facility and have had the capacity to support our customer 
requirements in a timely manner, something that we could not have said several years ago. In 
short, we have substantially improved service levels, which was a key request of our customers. 
Unfortunately, this has come at increased cost. The starting point for the DOS consolidation was 
from multiple data centers that were not able to provide the availability and scalability 
requirements that systems such as Passport and Visa require. 

We have also been able to exceed our virtualization goals: over 50% virtualized at ESOC East, 
35% virtualized across all our data centers. This will allow us to reap cost benefits in the future 
as we ramp up these numbers to the capacity of our new infrastructure. 

The primary challenge to achieving consolidation has been balancing resources between O&M 
and Consolidation: the more servers that we consolidate, the more we need to support. In a 
time of uncertain federal budgets, securing and prioritizing funding has been an ongoing issue. 
However, with senior management support we been able to fund our efforts as budgeted. 

We have accelerated one aspect of our program: since the FDCC Initiative began, we have 
initiated an overseas virtualization effort. The effort has allowed us to extend management 
efficiency benefits of virtualization that we have already been realizing domestically to our 
overseas posts. 

Regarding lessons learned, we have modified our tactical focus into a two pronged approach to 
consolidation, both of which we believe are critical for our ongoing success: 

• Customer outreach. Obtaining customer buy-in, determining customer capacity 
requirements, getting commitments to timelines, and managing customer expectations. 
This is a day-in-day-out effort which is crucial to the success of a program this broad with 
this many stakeholders. 

• Focus on data-center-by-data-center closure. We also have had focused efforts to close 
each legacy data center on a facility-by-facility basis. We believe that a directed effort for 
each data center is the only way accomplish closure of the room, otherwise day-to-day 
concerns and emerging customer requirements while drain the focus of the PMO team and 
kill the long-term schedule. 

6.2 Cost Savings 

The DOS remained within the FY11 budget for Data Center Consolidation and exceeded our cost 
avoidance goals (i.e. estimated costs avoided exceeded the projected costs avoided). Cost 
avoidance was primarily achieved through virtualization; it is estimated that the Department 
saved $1.5M in FY2011 by avoiding the hardware and power costs associated with the physical 
servers that did not have to be purchased or powered. We estimate that the Department will 
be able to reallocate over $5.8M per year once the transitions are complete and steady state 
has been achieved at the ESOCs. 

It is also estimated that the Department avoided $156K in operational costs by having turned 
down data centers in FY11. 

Our primary long-term (four to five years) cost goal is to maintain constant OPEX costs as we 
increase the number of systems being centrally supported. We have currently not calculated 
the total cost benefits to the Department of being able to guarantee highly available and secure 
service to our customers, though the intangible cost of service outages at our legacy data 
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centers in the past has been high. Over time we expect to explore ways of addressing the return 
on investment to continue offering a higher level of service. 

There were no significant unanticipated consolidation costs. The Department's enacted FY11 
budget fully funded the DCC effort. 
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